Audio-Orientated Immersive Experience of an Event

ABSTRACT

The exemplary embodiments are related to a device, system and method for implementing an audio orientated immersive experience of an event presented at a user device. A server is connected to a user device. The server generates a virtual environment that corresponds to a site of an event. The server receives tracking data corresponding to an object of the event. The tracking data is incorporated into the virtual environment. The server identifies a predetermined condition and selects enhancement data based on the predetermined condition. The predetermined condition corresponds to the tracking data and the virtual environment. The server transmits a data stream to the user device, the data stream including the enhancement data.

BACKGROUND INFORMATION

The audio included in the broadcast of an event, such as a sportingevent, may be produced by various sources (e.g., announcers, the crowd,the players, etc.). Under conventional circumstances, when audio outputis generated at a user device based on audio data included in abroadcast feed, the audio output may lack descriptive and immersivequalities. As a result, a user may have an unsatisfactory experiencewith the presentation of the event at the user device. Accordingly,there exists a need for a mechanism that enhances the audio output tocreate a more immersive experience of the event.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a system for providing an audio-orientated immersiveexperience according to various exemplary embodiments.

FIG. 2 shows an exemplary method for the enhancement server providing adata stream including enhancement data according to various exemplaryembodiments.

FIG. 3 shows an exemplary method for generating an audio-orientatedimmersive experience at a user device according to various exemplaryembodiments.

FIG. 4 shows an exemplary user device according to various exemplaryembodiments.

DETAILED DESCRIPTION

The exemplary embodiments may be further understood with reference tothe following description and the related appended drawings, whereinlike elements are provided with the same reference numerals. Theexemplary embodiments are related to devices, systems and methods forcreating an audio-orientated immersive experience of an event via a userdevice.

The exemplary embodiments are described with regard to a user devicethat is configured to generate audio output. The user device may also beconnected to one or more audio output accessory devices. For example,the user device may be a smartphone and the audio output accessorydevice may be headphones. However, any reference to the user devicebeing a particular type of electronic device or the audio outputaccessory device being a particular type of audio output device ismerely provided for illustrative purposes. The exemplary embodiments mayapply to any arrangement of one or more devices configured to generateaudio output.

The exemplary embodiments are also described with reference to thepresentation of a live sporting event at the user device. However, theexemplary embodiments are not limited to the presentation of a livesporting event. Throughout this description other examples will beprovided. It will be apparent to those skilled in the art that theconcepts described herein may be applicable to the presentation of audiodata corresponding to any type of real or virtual environment.

The broadcast feed for the event may include audio data representingvarious sources. For example, the audio data may include an announcerproviding commentary of the event, an announcer providing a play-by-playdescription, the crowd, the gameplay, the players, etc. Thus, the userdevice may generate audio output based on the audio data included in thebroadcast feed.

A server may be configured to provide the user device with enhancementdata. As will be described below, the enhancement data may enable theuser device to generate an audio-orientated immersive experience. Insome embodiments, the enhancement data may include audio data that mayprovide a descriptive indication of the actions that are occurringduring the event. In other embodiments, the enhancement data may includeaudio data configured to be generated as spatial audio output that makesthe user perceive audio output as originating from a particularlocation. The enhancement data may be selected or generated and thenprovided to the user device based on the occurrence of a predeterminedcondition associated with the event. Thus, the user device may generateaudio output based on broadcast audio data and audio output based on theenhancement data.

The audio-orientated immersive experience may enhance a visualrepresentation of the event. The audio orientated immersive experiencemay be configured to make the user feel as if the user was located atthe site of the live event or provide information conveying sounds. Theinformation conveying sounds may be realistic (e.g., the sound of abouncing basketball) or may not be realistic (e.g., cartoon like soundeffect). However, the audio-orientated immersive experience does notrely on a visual representation of the event and may be used as anindependent presentation of the event. Thus, while the exemplaryembodiments may provide a more impactful event experience for allscenarios, the exemplary embodiments may provide even more substantialbenefits in scenarios where a visual representation of the event may notbe available or feasible. For example, it may not be possible for avisually impaired user to have a satisfactory experience with theconventional presentation of the event and thus, the visually impaireduser may benefit from the descriptive and immersive qualities of theexemplary embodiments. In another example, a visual representation ofthe event may not be available to the driver of a car and thus, thedriver may benefit from the descriptive and immersive qualities of theexemplary embodiments. In another exemplary embodiment, the entityproviding the immersive audio experience may not have rights tobroadcast the video feed of the event. Accordingly, the exemplaryembodiments relate to creating an audio orientated immersive experiencethat makes the user feel as if the user was located at the site of thelive event, or provides the user additional information via an enhancedaudio experience, regardless of whether a visual representation of thelive event is provided.

Further, the exemplary embodiments utilize a universal design approachfor the audio-orientated immersive experience of the event. Universaldesign considers factors such as, equitable use, flexibility in use,simple and intuitive use, perceptible information, size and space forapproach and use, tolerance for error and low physical effort. Thus,while the exemplary embodiments may seem tailored towards visuallyimpaired users, the audio-orientated immersive experience may enhancethe event experience for any user.

To provide a general example of some of the aspects of theaudio-orientated immersive experience from the user perspective,consider the following exemplary scenario in which the event is abasketball game and the user is equipped with the user device andheadphones. In a first aspect, the server may select or generate audiodata (e.g., enhancement data) based on the occurrence of a predeterminedcondition. This audio data may be provided to the user device to serveas an indication of an action that occurs during the event. For example,an action that may occur during the basketball game is associated with aparticular type of audio output which indicates to the user a particulartype of action that occurred during the basketball game. As mentionedabove, the user device may receive both broadcast audio data andenhancement data. Thus, the enhancement data may be added to create amore descriptive and immersive experience for the user.

An exemplary sequence of actions may include dribbling from a firstlocation to a second location where an unsuccessful shot attempt hitsthe rim of a basketball hoop. In this example, dribbling may beassociated with audio output that includes sounds that may be perceivedas a basketball bouncing on the hardwood of the basketball court. Theunsuccessful shot attempt that hits the rim may be associated with audiooutput that includes a sound that may be perceived as a thud.Accordingly, when a player dribbles from a first location to a secondlocation on the basketball court, the audio output at the user'sheadphones includes sounds that may be perceived as a basketballbouncing on the hardwood of the basketball court and may be provide in amanner that indicates to the user the location and direction of travelof the ball. When the player performs an unsuccessful shot attempt thathits the rim, the audio output at the user's headphones includes soundsthat may be perceived as a thud. Accordingly, based on the audio outputcorresponding to the enhancement data, the user may be aware of theactions that are being performed by the players of the basketball game.

In a second aspect, audio output may be configured to be perceived bythe user as if the user were located at a particular location at thesite of the live event. For example, within the context of thebasketball game, the user may simulate being located where the midcourtline and the sideline intersect. Relative to this location, a firstbasketball hoop is located on the left side and a second basketball hoopis located on the right side. The audio output may be configured to beperceived by the user as if the user were located at this location. Inthe actual basketball game, as a player dribbles from the firstbasketball hoop towards the second basketball hoop, the audio output atthe user's headphones may initially be perceived on the user's left handside and then transitions across to the user's right hand side. When theplayer performs an unsuccessful shot attempt that hits the rim of thesecond basketball hoop, the audio output at the user's headphones isconfigured to be perceived as a thud originating from the spatialdirection of the second basketball hoop relative to the user's simulatedlocation on the basketball court (e.g., right hand side). Accordingly,the exemplary embodiments relate to creating a spatial experience thatsimulates being located at a particular location within the environmentof the event.

The above exemplary scenario is only provided as a general example ofsome of the aspects of the audio-orientated immersive experience fromthe user perspective and is not intended to limit the scope of theexemplary embodiments in any way. Further, as mentioned above, theexemplary embodiments are not limited to the presentation of a sportingevent and may be applicable to the presentation of audio datacorresponding to any type of real or virtual environment.

The exemplary embodiments will be described with regard to operationsthat are performed on the production side and operations that may beperformed on the user side. The operations on the production side mayrelate to generating and providing a data stream to a user device on theuser side. The data stream may enable the user device to provide anaudio-orientated immersive experience. The production side operationsmay be used in conjunction with other currently implemented streamingmechanisms, future implementations of streaming mechanisms orindependently from other streaming mechanisms. The operations on theuser side may relate to how the user device creates an audio-orientatedimmersive experience for the user. The user side operations may be usedin conjunction with other currently implemented techniques forpresenting audio data to a user, future implementations of techniquesfor presenting audio data to a user or independently from other similartechniques.

The user side and the production side may communicate with one anotherusing one or more networks. For example, the one or more networks mayinclude, but is not limited to, a cellular network, a wireless localarea network (WLAN), a personal area network (PAN), a peer-to-peer (p2p)network, a satellite or cable television distribution network, etc.Those skilled in the art will understand the type of connections,association procedures and communication protocols that may be used forcommunications between the user side and the production side.

FIG. 1 shows a system 100 for providing an audio-orientated immersiveexperience according to various exemplary embodiments. The system 100includes a user side 105 and a production side 150.

The user side 105 includes a user device 110, an audio output accessorydevice 112 and a haptic feedback accessory device 114. The user device110 may be any type of electronic component that is configured toreceive data via a network connection, e.g., a desktop computer, alaptop computer, a mobile phone, a tablet computer, a smartphone, asmart speaker, a phablet, an embedded device, a wearable device, anInternet of Things (IoT) device, a set-top box, a television, amultimedia receiver, a car's infotainment system, etc. Accordingly, theuser device 110 may represent any suitable electronic device.

The user device 110 may connect directly with the audio output accessorydevice 112 and the haptic feedback accessory device 114. The audiooutput accessory device 112 may be any type of electronic componentconfigured to connect to the user device 110 and generate audio output,e.g., headphones, a speaker, a wearable device, a further user device,etc. The haptic feedback accessory device 114 may be any type ofelectronic component configured to connect to the user device 110 andgenerate haptic feedback. Haptic feedback generally refers to usinghardware to create physical sensations to communicate with the user.FIG. 1 illustrates the audio output accessory device 112 and the hapticfeedback accessory device 114 as being external to the user device 110.However, in an actual system, the audio output accessory device 112 andthe haptic feedback accessory device 114 may be included in the userdevice 110.

Like the user device 110, the audio output accessory device 112 and thehaptic feedback accessory device 114 may also communicate directly withthe network. Reference to a single user device 110 connected toaccessory devices 112, 114 and associated with a single user is merelyprovided for illustrative purposes. An actual system may include anynumber of user devices, each connected to any number of accessorydevices, associated with any number of users.

The user device 110 and the accessory device 112, 114 may communicatewith one another using a wired connection. The user device 110 and theaccessory devices 112, 114 may also communicate with one another using ashort-range communication protocol, e.g., Bluetooth, WiFi, etc. Forinstance, if the user device 110 and the accessory devices 112, 114 arewithin a distance in which the short-range communication protocol may beused, the user device 110 and the accessory devices 112, 114 mayexchange data. Thus, the accessory devices 112, 114 may be configured towirelessly receive data utilizing the short-range communicationprotocol. In some scenarios, the user device 110 may relay data receivedfrom the network to the accessory devices 112, 114 over the short-rangecommunication pathway. In other scenarios, the user device 110 may relaydata received the accessory devices 112, 114 over the short-rangecommunication pathway to the network. The user device 110 and theaccessory devices 112, 114 may also communicate with each other usingtheir respective network connections.

In this example, the devices on the user side 105 may provide the userwith an audio-orientated immersive experience. As indicated above, whilethe audio-orientated immersive experience may enhance a visualrepresentation of the event, the visual representation of the event isnot required. The audio-orientated immersive experience may serve as anindependent presentation of the event.

The production side 150 may include an enhancement server 160 that mayinclude one or more communication interfaces to communicate with thecomponents on the production side 150 and the components on the userside 105. The enhancement server 160 may also include one or moreprocessors. The one or more processors may be represented as separateincorporated components of the enhancement server 160 or may be amodular component coupled to the enhancement server 160, e.g., anintegrated circuit with or without firmware. For example, the integratedcircuit may include input circuitry to receive signals and processingcircuitry to process the signals and other information.

The data that drives the audio-orientated immersive experience may beprovided to the user side 105, at least in part, by the enhancementserver 160. To provide an example, initially, the user device 110 mayestablish a connection with the enhancement server 160 via one or morenetwork connections. Subsequently, the enhancement server 160 mayprovide the user device 110 with one or more data streams. The datastreams may carry various different types of data including, but notlimited to, broadcast data and enhancement data. However, since theenhancement server may provide a data stream of only enhancement data,the enhancement data may be provided by an entity that does not own therights to broadcast the event. When the entity that does not own therights provides the enhancement data, the entity may also include nearreal-time information associated with the live event such as aplay-by-play data stream, graphic representation of game actions oranimations, etc.

Throughout this description “broadcast data” refers to multimedia datathat originates from a broadcast feed of the event. The multimedia datamay comprise one or more of video, audio, text, graphics, animations, aplay-by-play description, etc. In some embodiments, the enhancementserver 160 may receive broadcast data and incorporate the broadcast datainto a data stream. In other embodiments, the enhancement server 160 mayreceive broadcast data, modify the broadcast data and incorporate themodified broadcast data into a data stream. In further embodiments, theenhancement server 160 may not send any broadcast data to the userdevice 110. Instead, the user device 160 may receive broadcast datadirectly from a broadcast feed and use both the enhancement datareceived from the enhancement server 160 and the audio data receivedfrom the broadcast feeds to provide the audio-orientated immersiveexperience.

Throughout this description “enhancement data” may refer to a type ofdata that is generated or selected based on data corresponding to thelive event and is configured to cause the user device 110 to perform anoperation related to the audio-orientated immersive presentation of theevent. In this example, enhancement data may represent audio data, datarelated to generating haptic feedback, data that may be used to generateaudio output, data that may indicate to the user device 110 that audiodata stored locally at the user device 110 is to be used to generateaudio output or any other appropriate type of data.

The production side 150 may include a broadcast audiovisual feed 180 anda broadcast audio feed 182. The broadcast audiovisual feed 180 mayrepresent a combination of audio data and video data broadcast to thepublic (e.g., television broadcast, a live stream over the Internet,etc.). The broadcast audio feed 182 may represent audio data broadcastto the public (e.g., radio broadcast, a live stream the Internet, etc.).A person of ordinary skill in the art will understand how theaudiovisual feed 180 and the audio feed 182 may be delivered to the userside 105.

The enhancement server 160 may also receive the broadcast audiovisualfeed 180 and the broadcast audio feed 182. In a first aspect, thebroadcast data may be used by the enhancement server 160 to identify apredetermined condition corresponding to the event. Based on thepredetermined condition, enhancement data may be generated or selectedby the enhancement server and then provided to the user device 110. In asecond aspect, the enhancement server 160 may send modified orunmodified broadcast data to the user device 110. For example, theenhancement server 160 may provide a data stream that includes bothenhancement data and broadcast data to the user device 110. Thus, theuser device 110 may receive broadcast data (modified or unmodified) fromthe broadcast audio visual feed 180, the broadcast audio feed 182 fromthe enhancement server 160 or from any other appropriate source andenhancement data from the enhancement server 160.

Operations performed by the enhancement server 160 may be performedusing various different types of data received from various differentsources. In addition to the audiovisual broadcast feed 180 and thedescriptive audio feed 182, the data sources may include an ambientaudio feed 170, a descriptive audio feed 172, a video feed 174, ametadata feed 176 and an object tracking system 190. Thus, theenhancement server 160 may be equipped with a one or more communicationinterfaces to communicate with the user device 110, the audiovisualbroadcast feed 180, the descriptive audio feed 182, the feeds 170-176,the object tracking system 190 and any other component which maytransmit data to or receive data from the enhancement server 160.

The ambient audio feed 170 may include audio data that representsambient noise from the event such as, crowd noise, noise from theplaying field, the sidelines, the players, etc. The descriptive audiofeed 172 may include audio data that represents an audio description ofthe event such as, commentary, play-by-play analysis, a descriptionintended for an audio-only presentation, a description intended forvisually impaired users, etc. The ambient audio feed 170 and thedescriptive audio feed 172 may be analog or digital, recorded from asingle source (e.g., microphone, audio input device) or multiple sourcesand may be statically recorded or dynamically produced (e.g., by adirector of the live event). The ambient audio feed 170 and thedescriptive audio feed 172 may be delivered from the live event or aremote production facility to the enhancement server 160 by any meansknown in the art for delivering an audio feed.

The video feed 174 may include video data that represents the event. Thevideo data may be analog or digital and may be recorded by a singlesource (e.g., camera) or by multiple sources. When multiple sources areused to record the event, the video data may include video data from allof the sources or may be selectively switched among sources (e.g., by adirector of the event switching cameras). The video feed 174 may bedelivered from the live event or a remote production facility to theenhancement server 160 by any means known in the art for delivering avideo feed.

The metadata feed 176 may include metadata associated with the event. Inthe context of a sporting event, metadata may include event data such asgame score, time remaining, team statistics, individual playerstatistics, game situation, etc. The metadata feed 176 may be deliveredfrom the live event or a remote production facility to the enhancementserver 160 by any means known in the art for delivering a metadata.

One or more of the ambient audio feed 170, the descriptive audio feed172, the video feed 174 and the metadata feed 176 may be used tobroadcast the event. Thus, in certain scenarios, the enhancement server160, broadcast audiovisual feed 180 and the broadcast audio feed 182 allreceive data from feeds originating from the same location or entity.Additionally, reference to a separate ambient audio feed 170,descriptive audio feed 172, video feed 174 and metadata feed 176 is onlyprovided for illustrative purposes. Any two or more of the feeds 170-176may also be transmitted in a single feed.

The object tracking system 190 may collect tracking data associated withone or more objects of the event from any of a variety of sources. Theobjects may represent a person (e.g., player, referee, coach, etc.) orobject (e.g., equipment, ball, puck, etc.) located at the event site.The object tracking system 190 is shown as being external to theenhancement server 160, however, this is only for illustrative purposes.Throughout this description, various operations described with regard tothe object tracking system 190 may alternatively be performed at theenhancement server 160.

In some embodiments, the object tracking system 190 may collect thetracking data based on identifying and tracking objects using videodata. The video data may be provided by the broadcast audiovisual feed180, the video feed 174 or video data captured by a different source.The exemplary embodiments are not limited to any particular type oftechnique for tracking objects using video data. A person of ordinaryskill in the art would understand how objects may be tracked using videodata.

In other embodiments, the object tracking system 190 may collect thetracking data from sensors 192 located at the live event. There arevarious different types of sensors that may be used. For example, asensor may track an RFID chip located on an object (e.g., a player, auniform, equipment, the ball) to collect location data corresponding tothe object. However, the exemplary embodiments are not limited to anyparticular type of technique for tracking objects using a sensor. Aperson of ordinary skill in the art would understand how objects may betracked using sensor.

FIG. 2 shows an exemplary method 200 for the enhancement server 160providing a data stream including enhancement data according to variousexemplary embodiments. The method 200 will be described with regard tothe system 100 of FIG. 1.

In 205, the enhancement server 160 receives configuration information.The configuration information may relate to characteristics of thepresentation of the audio-orientated immersive experience.

In a first aspect, the configuration information may relate to the typeof audio output that is to be presented to the user to indicate theoccurrence of an action during the event. For example, continuing withthe basketball scenario provided above, the configuration informationmay indicate to the enhancement server 160 that audio data for a soundthat may be perceived as a basketball hitting the hardwood of the courtis to be provided to indicate that the basketball in the actual eventhits the hardwood of the court and audio data for a sound that may beperceived as a thud is to be provided to indicate that the basketball inthe actual event hits the rim. Further examples include, a made basketmay be associated with a sound that may be perceived as a swish, aplayer moving may be associated with a sound that may be perceived asshoes squeaking, a blocked shot may be associated with a sound that maybe perceived as a hand swatting the ball, etc. The type of audio outputis not required to be a realistic representation of the action in theevent and may be set to any type of sound including words or phrases,music, sound effects, etc. Any reference to a particular type of audiooutput is merely provided for illustrative purposes and is not intendedto limit the exemplary embodiments in any way. For each event, any typeof action may be associated with any type of audio output.

In a second aspect, the configuration information may relate to one ormore feeds (e.g., broadcast audiovisual feed 180, broadcast audio feed182, ambient audio feed 170, descriptive audio feed 172, video feed 174,etc.) that are to be included as part of the audio-orientated immersiveexperience. As will be explained below, this type of configurationinformation may indicate to the enhancement server 160 one or more feedsthat are to be forwarded to the user side 105 or how a feed is to bemodified before being sent to the user side 105. In some embodiments,the user side 110 may receive a feed from a source other than theenhancement server 160, e.g., broadcast audiovisual feed 180, broadcastaudio feed 182. Thus, the configuration information may also indicate tothe enhancement server 160 which feed the enhancement data may bepresented with at the user device 110 regardless of the entity that isto provide the feed to the user device 110.

In a third aspect, the configuration information may relate to anylocation at the event. This location may be any location at the site ofthe live event. For example, the user may select a seat at the venue ora location on the playing surface. This information may indicate to theenhancement server 160 that audio output should be perceived by the userrelative to this location. The enhancement server 160 may use thislocation information when generating enhancement data or when modifyinga feed of the event.

The configuration information may be received by the enhancement server160 as default settings, as user input on the production side or as userinput at the user device 110. To ensure that the user is aware of theaudio output that will be provided and has the option to alter theconfiguration information, an orientation procedure may be performed.During this procedure, the enhancement server 160 may describe to theuser at the user device 110 what the audio-oriented immersive experiencemay include. For instance, continuing with the basketball scenario, theenhancement server 160 may provide an audio clip to the user device 110that include the sound of a basketball hitting the hardwood of the courtand inform the user that this sound means a player is dribbling theball. Similarly, an audio clip may be provided that includes the soundof a thud and the enhancement server 160 may indicate to the user thatthis sound indicates that a player in the game has shot the ball and hitthe rim. In addition to the type of sounds the user may hear, theorientation procedure may include a training step for acclimating theuser to the spatial aspect of the audio orientated immersive experience.For example, the enhancement server 160 may provide an audio clip thatexplains that sounds that are perceived on the user's left side indicateaction near the home team's basket and sounds that are perceived on theuser's right side indicate action near the away teams basket. Duringoperation, an orientation procedure may be initiated in response to apredetermined event such as the end of a quarter or half. This mayensure the user is aware of any changes to the spatial aspect of theaudio orientated immersive experience caused by a change in the gamesuch as the home and away team's basket. The above examples are merelyprovided for illustrative purposes, the orientation procedure may beperformed in any appropriate manner.

In 210, the enhancement server 160 may generate a virtual environmentthat corresponds to the site of the event. For instance, continuing withthe basketball scenario, the enhancement server may generate a virtualrepresentation of the venue including the basketball court. The virtualenvironment may include a three-dimensional coordinate system (x,y,z).In this example, the coordinates (0, 0, 0) may represent a center pointon the surface of the basketball court. However, this is merely providedfor illustrative purposes, the exemplary embodiments may utilize anyappropriate coordinate system.

In some embodiments, the virtual environment may be modeled and renderedin 210. In other embodiments, the enhancement server 160 may have accessto a virtual environment model previously generated for the site of thelive event and may then render the virtual environment for theoperations performed in the method 200 using the model previouslygenerated.

In 215, the enhancement server 160 may receive tracking datacorresponding to one or more objects of the event. This tracking datamay be input into the virtual environment with three-dimensionalcoordinates. The three-dimensional coordinate system and thethree-dimensional coordinates may be used to determine the actions thatare occurring in the actual event. To provide an example, as a player inthe actual event dribbles from one end of the court to another, trackingdata corresponding to the player and tracking data corresponding to thebasketball may be collected and incorporated into the virtualenvironment. When the tracking data corresponding to the ball includes acoordinate associated with the surface of the basketball court (e.g.,y=0), this may indicate to the enhancement server 160 that in the actualevent the basketball has hit surface of the court. The corresponding xcoordinate of the tracking data may indicate where with the bounds ofthe court the ball has hit the surface. When the tracking datacorresponding to the player includes a first coordinate point at a firsttime (e.g., x=0) and a second coordinate point at a second time (e.g.,x=5), this may indicate to the enhancement server 160 that the playerhas moved from the center of the court towards a particular end of thecourt.

In 220, the enhancement server 160 determines whether a predeterminedcondition is satisfied. The predetermined condition may indicate to theenhancement server that enhancement data is to be provided to the userdevice 110. The enhancement server 160 may be configured to monitor forany of a variety of different types of predetermined conditions based onany of a variety of different factors. If the predetermined condition isnot satisfied, the method 200 returns to 215 to continue to monitor fora predetermined condition. If the predetermined condition is satisfied,the method 200 continues to 225.

In one example, the predetermined condition may be based on the trackingdata and the virtual environment. For instance, certain points on thecoordinate system may be associated with certain aspects of the site ofthe event. In this example, a y coordinate equal to 0 (or within acertain threshold to 0) may represent the surface of the court, variousx coordinates may indicate the perimeter of the court and a first set ofx, y, z coordinates may represent a first basketball hoop and a secondset of x, y, z coordinates may indicate the second basketball hoop. Whenthe tracking data of any object is incorporated into the virtualenvironment and corresponds to a coordinate that represents an aspect ofthe site of the live event, this may indicate that an action of interesthas occurred (e.g., the ball has hit the ground, gone out of bounds, hitthe rim, etc.) Thus, the enhancement server 160 may determine that audiodata corresponding to these types of actions is to be provided to theuser device 110.

The enhancement server 160 may also monitor for predetermined conditionscorresponding to a different type of data. For example, when event datashows that the score of a game is tied, this may indicate to theenhancement server 160 that enhancement data configured to triggerhaptic feed at the user device 160 is to be provided to the user device110. In another example, when the event data shows that the game clockor another similar temporal aspect of the game is at a particular state,the enhancement data may be configured to trigger haptic feedback orgenerate audio output that indicates to the user the state of thecorresponding temporal aspect (e.g., the game clock may soon expire,within two-minutes to the end of the quarter/half/game, etc.). Toprovide a further example, the enhancement data may be configured togenerate distinguishing audio output corresponding to a first team usinga first audio pitch and distinguishing audio output corresponding to asecond team using a second audio pitch. This may allow the user todifferentiate which team performed an action. For instance, continuingwith the basketball example, when the team that is losing makes a basketthe audio output (e.g., the sound of the swish) may be configured to beoutput at a low pitch and when the team that is winning makes a basketthe audio output may be configured to be output at a high pitch. As thelead increases, the difference in pitch may increase and as the leaddecreases the difference in pitch may decrease. This not only allows theuser to identify the team performing the action but may also indicatethe significance of a particular action on the actual game. The aboveexamples are provided for illustrative purposes, the exemplaryembodiments may apply to a predetermined condition that is based on anyappropriate type of data corresponding to the event.

In 225, the enhancement server 160 selects enhancement data. Theenhancement data may be based on the predetermined condition detected in220 and the configuration information received in 205. To provide anexample, the user has requested to simulate as if the user was locatedat a certain position at the live event. This configuration informationalong with the tracking data is incorporated into the virtualenvironment. When the tracking data indicates that an action isoccurring at a certain point within the virtual environment, an audioclip is selected (e.g., a thud, a swish, a cartoon sound effect, etc.)This audio clip may then be configured based on the user's simulatedlocation so that at the user device the audio output is perceived as ifthe user was located at the simulated location. Thus, if the home teammade a basket the audio clip may be perceived as if it was originatingfrom the side of the user corresponding to the home team's basket. Thisexample is merely provided for illustrative purposes and is not intendedto limit the exemplary embodiments in any way. As mentioned above, theenhancement data may represent audio data, data related to generatinghaptic feedback, data that may be used to generate audio output, datathat may indicate to the user device 110 that audio data stored locallyat the user device 110 is to be used to generate audio output or anyother appropriate type of data.

The enhancement data may be generated by the enhancement server 160,retrieved from a storage location on the production side 150, triggerthe user device 110 to generate audio output using audio data stored onthe user side 105 or may be audio data collected from the event and thenmodified. Continuing with the basketball scenario, the enhancement datamay be audio data corresponding to the identified action (e.g., dribble,hitting the rim, etc.). In some embodiments, the enhancement data is anaudio clip stored on the production side 150 and accessible by theenhancement server 160.

In other embodiments, the enhancement data is audio data collected fromthe event and configured to be perceived with a certain magnituderelative to the other audio presented to the user. That is, instead ofthe faint sound of a basketball hitting the court that may be perceivedin the conventional broadcast of an event, the sound of the basketballhitting the court is collected from the event and configured to beperceived at a greater magnitude such that the user can clearly hear theaudio indicative of the basketball hitting the hardwood.

The enhancement server 160 may configure the audio data that is toindicate that an action has occurred in the live event to be outputwithin a particular frequency range. The enhancement server 160 may alsoprevent any other audio data from using that frequency range. Thisincreases the likelihood that the user is able to perceive the audiooutput associated with an action in the live event over the other audiooutput included in the presentation of the event. For example, theenhancement server 160 may configure a set of audio data to be output ata first frequency range. The enhancement server 160 may configure otheraudio data to be output at a frequency range that is not within thefirst frequency range or may provide the user device 110 withinstructions regarding filtering other audio from being output withinthis first frequency range. Since the set of audio output is the onlyaudio output within the first frequency range the user may more clearlyperceive this audio output.

As indicated above, in some embodiments the enhancement data may beconfigured to generate haptic feedback at the user device 110 or thehaptic feedback accessory device 114. In other embodiments, theenhancement data may be spatial audio data configured to be perceived asif the user was located at the location indicated in the configurationinformation of 205. Thus, the enhancement data may represent any type ofdata sent to the user device 110 that is intended to enhance the eventexperience for the user.

In 230, the enhancement server 160 sends a data stream includingenhancement data to the user device 110. The enhancement data may besynchronized with other audio or video data that is included in thisdata stream, another data stream provided to the user device 110 by theenhancement server 160 or with a feed provided from different source.Synchronization may be performed on the production side 150 or mayinclude data that enables the user device 110 to synchronize theenhancement data with other data from the event.

During the broadcast of the event, there may be scenarios in which thereis no action of interest occurring at the site of the event and theavailable broadcast data is limited. For example, conventionally, when areplay is presented to the user side during the broadcast of the event,there may be no action of interest occurring at the site of the eventand the only audio provided may be of announcers commenting on thereplay. This conventional presentation may rely on the visualrepresentation provided in the broadcast. To ensure that these types ofscenarios do not cause an unsatisfactory audio-orientated immersiveexperience, the enhancement server 160 may be configured to provideenhancement data during a replay. This may include, identifying theonset or occurrence of a replay, retrieving audio data related to theaction (e.g., enhancement data, ambient audio, descriptive audio, etc.)that is the subject of the replay and providing it to the user device110.

To ensure that the user knows the context of the event, enhancement datamay include audio data that describes the current context of the event.For example, this audio data may include the time remaining, the score,the players on the court or any other appropriate context information.This audio data may be provided to the user device 110 based on aschedule, a timer, the occurrence of a predetermined condition at theevent, user input on the production side 150, user input on the userside 105, etc. During operation, the enhancement server may generatethis audio data using the event data received from the metadata feed176, the broadcast audiovisual feed 180 or any other appropriate source.

FIG. 3 shows an exemplary method 300 for generating an audio-orientatedimmersive experience at a user device 110 according to various exemplaryembodiments. The method 300 will be described with regard to the system100 of FIG. 1.

In 305, the user device 110 establishes a connection with the audiooutput accessory device 112. As mentioned above, in some embodiments,the audio output accessory device 112 may be headphones. In otherembodiments, the audio output accessory device 112 may be a surroundsound system. The exemplary embodiments are not limited to anyparticular audio output device and may apply to any arrangement of oneor more devices configured to generate audio output.

In 310, the user device 110 establishes a connection with theenhancement server 160. For example, the user device 110 may utilize auser-facing application or a web browser to access an Internet protocol(IP) address for the enhancement server. Those skilled in the art willunderstand the type of connections that may facilitate communicationbetween the user device 110 and the enhancement server 160.

In 315, the user device 110 sends location information (e.g., thelocation in the venue from which the user would like to experience theevent) to the enhancement server 160. As mentioned above, the locationinformation may be provided to the enhancement server 160 asconfiguration information during the orientation procedure. However, theexemplary embodiments are not limited to this scenario and may apply tothe user device 110 providing the location information to theenhancement server 160 in any appropriate manner.

In 320, the user device 110 receives a data stream from the enhancementserver 160. The data stream may include enhancement data. In someembodiments, the data stream may also include broadcast data or modifiedbroadcast data. In other embodiments, the user device 110 receives abroadcast feed from a different source, e.g., broadcast audiovisual feed180, broadcast audio feed 182, etc. The enhancement data may besynchronized with the broadcast data or the modified broadcast data onthe production side 150 prior to it being received by the user device110 or the user device 110 may be configured to synchronize theenhancement data with other data.

In 325, the user device 110 or the audio output accessory device 112generates audio output associated with the enhancement data. Thus, theuser device 110 may receive the enhancement data from the enhancementserver 160 and forward it to the audio output accessory device 112 via ashort-range communication protocol. Audio output is then generated bythe accessory device 112. The audio output may be configured to beperceived spatially, as if the user was located at the locationindicated in the location information.

The exemplary embodiments are not limited to a basketball game. Theenhancement data may be used to generate an audio-orientated immersiveexperience for the presentation of any type of multimedia data. Forexample, the enhancement data may be used to provide a more impactfulevent experience associated with a dance routine or competition. Theenhancement data in this scenario may trigger audio output that isintended to indicate to the user when the dancer's feet hit the ground.In another example, the enhancement data may accompany an audio book.The enhancement data in this scenario may be configured to indicate tothe user the actions that are being described in the book or location ofdifferent speakers to create a more impactful audio book presentation.

The user device 110 may also be connected to a haptic feedback accessorydevice 114. Based on the enhancement data, the user device 110 may causethe haptic feedback accessory device 114 to generate haptic feed. Like,the audio output accessory device 112, the haptic feedback accessorydevice 114 may receive data from the user device 100 via the short-rangecommunication protocol. To provide an example, the haptic feedbackaccessory device 114 may be a board that is configured to be held by theuser. The board may generate haptic feedback using motors, water jets orany other appropriate hardware. The haptic feedback may provide the userwith a physical sensation that describes what is happening in the liveevent. For example, continuing with the basketball scenario, hapticfeedback may be generated when the crowd noise at the event reaches acertain threshold, when players collide, when a basket is made or when aball hits the rim. There may also be haptic feedback that corresponds tomovement of one or more object at the event. Thus, the user may be ableto feel where the players are located on the court and where they aremoving on the court based on the haptic feedback accessory device 114.

FIG. 4 shows an exemplary user device 110 according to various exemplaryembodiments. Those skilled in the art will understand that the userdevice 110 may be any type of electronic component that is configured toexecute software or firmware, e.g., a desktop computer, a mobile phone,a tablet computer, a smartphone, phablets, embedded devices, wearabledevice, Internet of Things (IoT) devices, etc. Accordingly, the userdevice 110 may represent any electronic device.

The user device 110 may include a processor 415, a display device 420,an input/output (I/O) device 425, a transceiver 430, an audio outputdevice 435 and other components 440. The audio output device 435 mayrepresent one or more components configured to generate audio output,e.g., one or more speakers, headphones, etc. In some embodiments, theaudio output device 345 may be a component internal to the user device110. In other embodiments, the audio output device 345 may be externalto the user device 110 and connected via a wired or wireless connection.Those skilled in the art would understand the procedures and protocolsthat may be utilized by the audio output device 345 and the user device110 to establish and maintain a connection. The other components 440 mayinclude, for example, an audio input device, a battery, a dataacquisition device, ports to electrically connect the user device 110 toother electronic devices, ports that provide a wired connection to aninternet protocol (IP) based network, hardware configured to generatehaptic feedback, etc.

The processor 415 may be configured to execute a plurality ofapplications of the user device 110. For example, the applications mayinclude an audio-orientated immersive experience engine 445. In a firstaspect, the audio-orientated immersive experience engine 445 may managecommunications with the enhancement server 160. In a second aspect, theaudio-orientated immersive experience engine 445 may manage how theenhancement data and broadcast data is presented to the user. In someembodiments, the operations described above with regard to theaudio-orientated immersive experience engine 445 may be performed on thenetwork side (e.g., the enhancement server 160). In this type ofconfiguration, the user device 110 may receive data and information thatis already configured to be generated as output at the user device 110or any connected accessory devices.

The audio-orientated immersive experience engine 445 being executed bythe processor 415 is only exemplary. The functionality associated withthe audio-orientated immersive experience engine 445 may also berepresented as separate incorporated components of the user device 110or may be a modular component coupled to the user device 110, e.g., anintegrated circuit with or without firmware. For example, the integratedcircuit may include input circuitry to receive signals and processingcircuitry to process the signals and other information. In addition, insome user devices, the functionality described for the processor 415 issplit among two or more processors. The exemplary embodiments may beimplemented in any of these or other configurations of a user device.

The display device 420 may be a hardware component configured to displaydata or other information to a user while the I/O device 425 may be ahardware component that enables the user to enter inputs. The displaydevice 420 and the I/O device 425 may be separate components orintegrated together such as a touchscreen. The transceiver 430 may be ahardware component configured to establish a connection with a wirelessnetwork.

Those skilled in the art will understand that the above-describedexemplary embodiments may be implemented in any suitable software orhardware configuration or combination thereof. An exemplary hardwareplatform for implementing the exemplary embodiments may include, forexample, an Intel x86 based platform with compatible operating system, aWindows platform, a Mac platform and MAC OS, a Linux based OS, a mobiledevice having an operating system such as iOS, Android, etc. In afurther example, the exemplary embodiments of the above described methodmay be embodied as a computer program product containing lines of codestored on a computer readable storage medium that may be executed on aprocessor or microprocessor. The storage medium may be, for example, alocal or remote data repository compatible or formatted for use with theabove noted operating systems using any storage operation.

It will be apparent to those skilled in the art that variousmodifications may be made in the present disclosure, without departingfrom the spirit or the scope of the disclosure. Thus, it is intendedthat the present disclosure cover modifications and variations of thisdisclosure provided they come within the scope of the appended claimsand their equivalent.

1-20. (canceled)
 21. A method, comprising: at an enhancement serverconnected to a user device: generating a three-dimensional model of avirtual environment that corresponds to a site of an event, wherein thethree-dimensional model of the virtual environment includes athree-dimensional coordinate system; receiving configuration informationcorresponding to characteristics of the event; receiving tracking datacorresponding to an object of the event, the tracking data of the objectbeing independent of the location of the user device; inputting thetracking data into the three-dimensional model of the virtualenvironment; identifying a predetermined condition within thethree-dimensional model of the virtual environment based on at least thethree-dimensional coordinate system and the tracking data; selectingenhancement data based on the predetermined condition and theconfiguration information; and transmitting a data stream to the userdevice, the data stream including the enhancement data.
 22. The methodof claim 21, wherein the configuration information comprises datarelated to a type of audio output to be presented to a user to indicatean occurrence of an action corresponding to the predetermined condition.23. The method of claim 21, wherein the configuration informationcomprises data related to one of a broadcast audiovisual feed, abroadcast audio feed, an ambient audio feed, a descriptive audio feed, avideo feed, or a metadata feed.
 24. The method of claim 21, wherein theconfiguration information comprises data related to a location at thesite of the event.
 25. The method of claim 24, wherein the location isone of in an area of the site where an action corresponding to thepredetermined condition occurs or not in an area of the site where anaction corresponding to the predetermined condition occurs.
 26. Themethod of claim 21, wherein the configuration information is received asa default setting or as user input from a user of the user device or aproduction user related to an entity that is producing an audio or videobroadcast of the event.
 27. The method of claim 21, further comprising:sending, prior to receiving configuration information from a user of theuser device, orientation information related to the configurationinformation.
 28. The method of claim 27, wherein the orientationinformation comprises an audio clip related to the enhancement data or aspatial audio data related to the enhancement data.
 29. The method ofclaim 21, wherein the event is a sporting event and the site is a venuefor the sporting event.
 30. The method of claim 21, wherein theenhancement data is presented in a live stream of the event or duringreplays that are interspersed in the live stream of the event.
 31. Themethod of claim 30, wherein, when the enhancement data is presentedduring a replay, the user device is notified that the presentation isthe replay.
 32. An enhancement server, comprising: a communicationinterface configured to communicate with a user device; and a processorcommunicatively coupled to the communication interface and configured toperform operations comprising: generating a three-dimensional model of avirtual environment that corresponds to a site of an event, wherein thethree-dimensional model of the virtual environment includes athree-dimensional coordinate system; receiving configuration informationcorresponding to characteristics of the event; receiving tracking datacorresponding to an object of the event, the tracking data of the objectbeing independent of the location of the user device; inputting thetracking data into the three-dimensional model of the virtualenvironment; identifying a predetermined condition within thethree-dimensional model of the virtual environment based on at least thethree-dimensional coordinate system and the tracking data; selectingenhancement data based on the predetermined condition and theconfiguration information; and transmitting a data stream to the userdevice, the data stream including the enhancement data.
 33. Theenhancement server of claim 32, wherein the predetermined condition isbased on at least i) the three-dimensional coordinate system and ii) thetracking data.
 34. The enhancement server of claim 32, wherein theenhancement data is audio data that is configured to generate audiooutput that indicates an action that occurred during the event.
 35. Theenhancement server of claim 32, wherein the enhancement data isconfigured to cause the user device to generate haptic feedback.
 36. Theenhancement server of claim 32, wherein the operations further comprise:receiving an indication of a location corresponding to the event,wherein the enhancement data is spatial audio data that is configured togenerate audio output that is to be perceived spatially relative to thelocation.
 37. The enhancement server of claim 32, further comprising:receiving a request for event data from the user device, wherein theenhancement data includes audio data that is configured to generateaudio output that includes a description of the event data.
 38. Acomputer program product comprising a set of instructions to be executedby a user device to cause the user device to perform operations,comprising: transmitting location information to the enhancement server,the location information corresponding to a location at a site of anevent, wherein the site of an event corresponds to a three-dimensionalcoordinate system included in a three-dimensional model of a virtualenvironment rendered by the enhancement server; receiving a data streamfrom the enhancement server, the data stream including enhancement dataassociated with the location information and configuration informationrelated to characteristics of the event; and generating audio outputbased on the enhancement data, the audio output configured to beperceived spatially relative to the location at the site of the event.39. The computer program product of claim 38, wherein the configurationinformation comprises data related to a location at the site of theevent received as user input from a user of the user device.
 40. Thecomputer program product of claim 38, wherein generating the audiooutput includes transmitting the audio data to an accessory device via ashort-range communication protocol.